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(54) Pointing to displayed objects in a computer system 

(57) An object to be pointed to in a user's computer 
display (109) is indicated with an animated icon (113). 
In a computer network, such as the Worldwide Web, a 
pointer is transmitted from one user to another user 
(1 14) via an improved protocol. A graphical design (201) 
of a computer icon is also disclosed. 
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Description 

The present invention relates to computer systems 
and their operation. More specifically, but not exclu- 
sively to a computer system in which objects on a com- 
puter display are indicated with an improved pointer. 

Various pointer systems for computer displays are 
well Known to those of skill in the art. Most commonly, 
such pointers take the form of arrows, l-bars and other 
such elements. In the Windows environment, for exam- 
ple, such pointers are moved from one location to 
another by detecting a series of "mouse move" signals 
from a computer pointing device such as a mouse. Gen- 
erally, the computer system monitors inputs from a 
mouse or similar pointing device, and in a roughly con- 
tinuous manner, moves a pointer from one screen loca- 
tion to another. The pointer may be used to point to a 
variety of screen objects such as buttons, cursor bars, 
and a wide range of other screen objects. 

In some situations it is desirable for a user to point 
to an object on a computer display that is on another 
user's screen. As a simple example, suppose that a 
user is remotely advising another user regarding a Web 
page design. Commonly, the first user would simply 
advise the second user by verbal communication such 
as through a telephone call or even over an internet 
voice link. It is often awkward or extremely difficult for 
one user to direct another's attention to a specific loca- 
tion on the screen display. 

Solutions have been proposed to this problem. For 
example, remote pointing is supported by products such 
as telepointers. Such telepointers may be used to point 
to an object in a screen display, sometimes taking the 
form of a WYSIWIS (What You See Is What I See) dis- 
play. In some cases the displays are maintained in a 
concurrent state by way of a controller/controllee rela- 
tionship between the two displays. In such situations, 
the user at a controller display moves his or her pointing 
device, and the pointer on all of the displays of the other 
users is moved in unison. This enables one user to point 
to objects and allow others to see where the user is 
pointing. Such products include ShowMe SharedApps® 
from Sun Microsystems®, Inc. See also Stefik et al., 
"WYSIWIS Revised: Early Experiences with Multi-User 
Interfaces," Conference o n Computer Supported Coop- 
erative Work. December 3-5, 1986. 

While meeting with some success, such systems 
have also met with certain limitations. For example, in 
some systems users are not using the same system for 
rendering their respective displays on their computers. 
A common example of this situation is found in internet 
browsers where users each select their own internet 
browsing tool, font styles and colors, CPUs, operating 
systems, and monitors, often with the result that remote 
pointers cannot communicate with each individual user. 
Systems such as the Worldwide Web (WWW) are flexi- 
ble through the use of a common HTML image trans- 
mission standard, but do not effectively support remote 
pointing. Further, the communication bandwidth that 



would be required to operate such systems using con- 
ventional controller/controllee techniques would create 
great burden on any network system. Such problems 
would be particularly apparent when users are in differ- 

s ent countries, especially if some users have poor con- 
nections to the WWW. 

From the above it is seen that improved systems 
and methods for pointing to objects on computer 
screens are desirable, especially for use in computer 

10 networks. 

Improved systems and methods for pointing to 
objects on a computer display can be provided by 
embodiments of the present invention. According to one 
specific embodiment of the invention, upon pointing to 

is an object, a pointer icon changes appearance. An illus- 
tration of such changing appearance is a "raindrop" icon 
wherein an illustration of a drop of water appears in the 
display and expands over time. 

Embodiments may also include a system for point- 

20 ing to objects in a computer network. Pointer objects are 
embedded in a serial data stream when a first user 
points to an object. When the serial data stream is 
assembled on a second user's screen, the display 
includes a pointer at or near the location where the first 

25 user has pointed. An important application of the inven- 
tion will be found in the Worldwide Web wherein point- 
ers are embedded in HTML bit streams of images. 

Accordingly, one specific embodiment provides a 
method of operating a computer including the steps of 

30 moving a pointer from one object to another object on a 
computer screen with a user input device; receiving 
user input for pointing to one of the objects on the com- 
puter screen; and changing an appearance of a pointing 
icon in response to the step of receiving user input for 

35 pointing to one of the objects. 

An embodiment may also include a method for 
pointing to objects in a computer network including the 
steps of transmitting screen objects across the compu- 
ter network in a serial form for display on a first user 

40 screen; receiving pointing input from a second user, the 
pointing input defining a location on the screen where 
the second user desires to point; inputting pointing 
objects into the serial stream; and displaying pointer 
graphics on said first user screen corresponding to said 

45 location where said second user desires to point 

For a better understanding of the invention, and to 
show how the same may be carried into effect, refer- 
ence will now be made, by way of example, to the 
accompanying drawings, in which :- 

50 

Fig. 1 is an overall block diagram of a computer sys- 
tem according to one embodiment of the invention 
herein; 

Figs. 2A to 2G illustrate a dynamic pointer to an 
55 object on a computer display; the background of the 
screen and the computer itself are illustrated with 
broken lines only for purposes of showing the envi- 
ronment of the dynamic pointer and not for the pur- 
pose of illustrating the invention herein; 
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Fig. 3 illustrates a method for displaying a pointer 
icon on a computer display of a remote user; 
Fig. 4 illustrates a method for generating a particu- 
lar temporally varying pointer icon; and 
Fig. 5 illustrates a method for transmitting a pointer 
icon from one user to another user in a computer 
network, using the Worldwide Wet as an example. 

Fig. 1 illustrates a computer network according to 
one embodiment of the invention. A computer 101 is 
provided with a central processing unit 103 and memory 
105. Memory 105 may include, for example, RAM and a 
permanent storage media such as a magnetic and/or 
optical disks in which the software for generating point- 
ers is stored (as well as other software). The computer 
101 is coupled to one or more user input devices 107. 
User input devices 107 will typically include a pointing 
device such as a mouse and a keyboard. The computer 
provides output on display 109. The display will nor- 
mally include one or more screen objects such as but- 
tons, regions for user input, pulldown menus, sliders, 
and other objects. A pointer 111 is overlaid on the 
screen display and can be moved from one location to 
another on the screen by manipulation of one or more of 
the user input devices. Images may be generated on the 
display via one of a variety of systems such as via a 
Web page. One preferred browser used for this purpose 
is HotJava, a browser available from Sun Microsystems. 

When the user has moved the screen icon to a 
desired location on the screen, the user provides input 
indicating a pointing operation. In some embodiments 
the pointing operation will be performed via a mouse 
click or pressing a designated key on the keyboard. In a 
particular embodiment the operation will be performed 
using a two button mouse in which the left mouse button 
is used for more conventional input such as pressing on 
displayed buttons or pointer bars, while the right or alter- 
nate mouse button is operation. 

When the user indicates that he/she desires to 
point to a particular object on the screen, a highlighted 
pointing operation may take place on the user's screen. 
Preferably, the highlighted pointing operation is con- 
ducted by displaying a highlight pointer icon 113 that 
changes temporally such that it will attract the attention 
of users more clearly. In some instances the highlighting 
pointer will be displayed for not only the user originating 
the pointing operation, but also others. In such cases, a 
remote computer system 114 linked by a network con- 
nection 1 1 5 is provided with a similar pointer 1 1 7 on its 
display. In these cases, one user may point to an object 
on another users screen with ease. In some instances 
(such as when a computer is used to generate a pres- 
entation) the pointer is displayed only on that particular 
user's display(s). In a preferred embodiment, each user 
is viewing a Web page generated from an HTML data 
stream. Each user will have his/her own browser soft- 
ware, which may or may not be the same browser soft- 
ware. 

Figs. 2A to 2G illustrate the graphical icon design 



utilized in a preferred embodiment of the invention for 
pointing to a screen object. A screen 200 and computer 
200A are illustrated with broken lines in Figs. 2A-2G 
only for purposes of showing the environment of the 

5 dynamic pointer and not for the purpose of the invention 
herein. As shown in Fig. 2A a pointer 201 is displayed 
with an appearance similar to that of a drop of water tell- 
ing on the screen. Figs. 2B to 2G illustrate the temporal 
variations in the appearance of the pointer icon. Fig. 2B 

10 illustrates a portion of the screen before the pointer is 
activated. Fig. 2C illustrates the icon in Hs initial form. As 
shown, a circular region 203 of "white space" on the 
screen is developed. In Fig. 2D the circular region 203 is 
surrounded by a dark region 205. In Fig. 2E the region 

75 205 is surrounded by a darker region 207. In Fig. 2F the 
initially displayed white region 203 has been removed. 
In Fig. 2G the darker region 205 has been removed. 
Subsequent to the display in Fig. 2G the screen returns 
to its initial form as shown in Fig. 28. 

20 Fig. 3 illustrates the overall methodology utilized to 
transmit a pointer from one user to another user accord- 
ing to a preferred embodiment of the invention. At step 
301 the system (which may include, for example, the 
computers of the two users and intervening network 

25 computers) inputs a pointing action from a user via, for 
example, a button push on a mouse, input from a touch 
screen display, or from another user input device. At 
step 303 the system (application program, generating 
system, and/or hardware) identifies a closest character 

30 or object on the screen adjacent to the location where 
the pointing action has taken place. In the HTML con- 
text, this step includes identifying the position in the 
HTML stream corresponding to this closest character or 
object. In the event that the user has the pointer 111 

35 directed to a specific character or screen object, this 
object will be chosen as a marked object. In cases 
where the user has pointed to a location on the screen 
where no object is displayed, the system will determine 
the distance to a plurality of adjacent screen objects, 

40 and select the object closest to the pointer location 
when the pointing operation has taken place. For exam- 
ple, in the HTML data format of the WWW, the screen 
location is identified to the system when a user enters a 
button click operation. The system then calculates the 

45 distance to each of a plurality of adjacent characters 
that have been generated by virtue of the HTML gener- 
ated display. The pointer icon image(s) are then 
inserted before or after this character in the HTML data 
stream. 

so At step 305 the system inserts a pointer element 
into a serial bit stream for transmission of an image to a 
remote user. A pointer element may refer to an image, 
object, or even a single indication (like a bold or italic 
mark in HTML) that indicates a pointer should be dis- 
ss played. One approach is to provide an extension to the 
HTML standard that includes a special formatting code 
that indicates a pointer should be displayed. In some 
embodiments, the remote user already has a display 
painted on his/her screen and the system transmits only 
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a series of characters indicating where the existing dis- 
play is to be changed. At step 307 the bit stream with 
the pointer element is transmitted to the remote user. At 
step 309 the remote user's computer system displays 
the screen image with the pointer icon in place. 5 

In some embodiments the pointer icon's image will 
vary temporally. Accordingly, in such embodiments the 
system will perform a test at step 31 1 to determine if the 
pointer display operation is complete. If not. the system 
will return to step 305 wherein a revised screen image 
will be generated, providing the user with the perception 
that the pointer icon is changing with time (the example 
above being in the form of a drop of water falling on the 
screen). In other embodiments, the transmission of a 
single pointer character or element will cause the 
remote user's system to generate the changing pointer 
image locally. For example, a pointer indicator may be 
inserted in the HTML data stream and the local browser 
may be programmed to paint a changing icon upon 
receipt of the pointer indicator. 

Fig. 4 illustrates a process for generating a tempo- 
rally varying pointer icon in a particular case of a water 
drop-type of image. As shown, the system first gener- 
ates a white circle 3 pixels in diameter centered at the 
location where pointing is desired at step 401. At step 
403 the system draws an open 50% grey circle sur- 
rounding the white circle. At step 405 the system draws 
an open black circle surrounding the grey circle. At step 
407 the system removes the white circle, and at steps 
409 and 41 1 the system removes the grey and black cir- 
cles, respectively. The image displayed according to this 
particular embodiment of the invention will have the 
appearance of a drop of water falling on the screen. 

Preferably, the elements displayed according to Fig. 
4 are displayed over a period of about 0.5 seconds 
each, and are removed from the screen in another 
group of 0.5 second intervals each. Therefore, the entire 
pointing process takes place over a time span of more 
than 1 second, preferably more than about 2 seconds, 
and most preferably more than 2.5 seconds, but prefer- 
ably less than 5 seconds. This is enough time to draw 
the user's attention to the screen location without signif- 
icantly obscuring the display. 

Fig. 5 illustrates the process of generating a pointer 
icon remotely in greater detail, using the Worldwide 
Web as an illustration with the known convention of the 
HTML transmission format. The transmission of images 
over the Worldwide Web is well understood to those of 
skill Ln the art, and is described in some detail in Morris, 
HTML For Fun and Profit. SunSoft Press, 1995, incor- 
porated herein by reference for all purposes. 

At step 501 the system displays a Web page com- 
mon to both a local and a remote user, or the local user 
controls the remote user Web page. At step 503 the 
system checks to determine if the same Web page is 
displayed at both the local and remote user's location. If 
not, an error message is displayed. 

At step 505 the system inputs the remote IP 
address of the remote user. At step 507 the system 



optionally activates the pointer system (assuming it is 
not always active). At step 509 the system inputs a 
remote pointer action such as a mouse click. At step 
51 1 the system selects a character position in the HTML 
data file for the current Web page closest to that of the 
pointer. At step 515 the pointer is transmitted and drawn 
on the remote user display. 

The HTML format is known to those of skill in the 
art. Generically, a Web page includes a stream of bits 
including a header and title, comments, a body, and an 
address. Text is marked with various formatting charac- 
ters such as paragraph format indicators, character for- 
mat indicators, and list format indicators. Image, sound, 
and animated video may be inserted into the bit stream 
according to recognized standards. According to a pre- 
ferred embodiment herein, after selection of a location 
for the pointer icon, an image of the first "part" of a tem- 
porally varying pointer icon is inserted into the HTML bit 
stream as, eg., a GIFF, TIFF, or JPEG bitmap, using an 
image or (IMG ) tag. Preferably, transparent cells are uti- 
lized such that the pointer icon may be overlaid on the 
object that is of interest. In a transparent display, the 
darkness of the previous screen display and the screen 
display after the addition of the pointer icon are consid- 
ered. If a pixel in the "underlying" image is dark, it 
makes that pixel darker when the "overlying" pointer 
icon is added. 

The above description is illustrative and not restric- 
tive. Many variations of the invention will become appar- 
ent to those of skill in the art upon review of this 
disclosure. Merely by way of example, while the inven- 
tion has been illustrated primarily with regard to gener- 
ation of "raindrop" pointers over the Worldwide Web, the 
invention is not so limited. 

Claims 

1. A method of operating a computer comprising the 
steps of: 

displaying a graphical display on a computer 
screen; 

moving a pointer from one object to another 
object on said computer screen with a user 
input device; 

receiving a first user's input for pointing to one 
of said objects on said computer screen; and 
temporally varying an appearance of a pointing 
icon in response to said step of receiving said 
first user's input for pointing to one of said 
objects. 

2. The method as recited in claim 1 further comprising 
the steps of: 

transmitting a pointer signal to a second user, 
wherein said step of temporally varying an 
appearance of said pointing icon is performed 
on at least said second user's computer 
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screen. 

3. The method as recited in claim 1 or 2 wherein said 
step of temporally varying comprises displaying a 
pointer having an appearance similar to a drop of 
water falling on said screen. 

4. The method as recited in claim 1 or 2 wherein said 
step of temporally varying comprises terminating 
display of said pointer icon after a period of time. 

5. The method as recited in claim 2, 3 or 4 wherein 
said step of temporally varying an appearance is 
performed on said first user's computer screen. 

6. The method as recited in any one of claims 1 to 5 
wherein said graphical display is generated over a 
computer network on at least said first user's com- 
puter screen and a second user's computer screen. 

7. The method as recited in claim 6 wherein said 
graphical display is a Web page. 

8. The method as recited in any one of claims 2 to 7, 
wherein said step of transmitting a pointer signal 
comprises the steps of: 

identifying an object in said graphical user 
interface closest to a location of a pointer input; 
and 

inserting a pointing indicator into a serial bit 
stream to be transmitted to said second user. 

9. The method as recited in claim 8 wherein said 
serial bit stream is an HTML bit stream. 

10. A method for pointing to objects in a computer net- 
work comprising the steps of: 

transmitting screen objects across said compu- 
ter network in a serial data stream for display 
on a first user screen; 

receiving pointing input from a second user, 
said pointing input defining a location on said 
first user screen where said second user 
desires to point; 

inputting pointing objects into said serial data 
stream; and 

displaying pointer graphics on said first user 
screen corresponding to said location where 
said second user desires to point. 

11. The method as recited in claim 10 wherein said 
serial data stream is an HTML data stream. 

12. The method as recited in claim 10 or 11 wherein 
said step of inputting pointer objects into said serial 
data stream comprises the steps of: 



identifying a screen display object closest to 
said location where said second user desires to 
point; and 

placing said pointing objects into said serial 
5 data stream at a position in correspondence 

with said identified screen display object. 

13. The method as recited in claim 10, 1 1 or 12 wherein 
said of displaying pointer graphics comprises the 

10 step of temporally varying an appearance of said 
pointer graphics on said first user screen. 

14. A computer system comprising: 

is a computer network for generating screen dis- 

plays on a plurality of user computers, the net- 
work including means for transmitting serial 
data streams to a plurality of user computers 
and means for generating screen displays 

20 based on said serial data streams; and 

a pointing system comprising a pointer inser- 
tion element for inserting pointers into said 
serial data stream. 

25 15. A computer system as recited in claim 14 wherein 
said system for transmitting serial data streams is a 
system for transmitting HTML data streams. 

16. A computer system as recited in claim 14 or 15 
30 wherein said pointing system generates temporally 

varying pointer icons. 

17. A computer system as recited in claim 14, 15 or 16 
wherein said pointing system identifies a closest 

35 screen object to a pointer, and said insertion ele- 
ment selects said closest screen object in said 
serial data stream for insertion of said pointer ele- 
ment. 

40 18. A graphical design for a temporally varying pointing 
icon in a computer system as shown in Figs. 2B to 
2G. 

19. A storage medium conveying software comprising: 

45 

software for generating a pointer in a remote 
display and; 

software for temporally varying an appearance 
of said pointer, whereby a screen location is 
so emphasized. 

20. A medium as recited in claim 19 wherein said soft- 
ware for changing an appearance of said pointer 
comprises software for inserting a pointer indicator 

55 in an HTML data stream. 

21. A medium as recited in claim 20 further conveying 
software for identifying a location of a nearest 
object to a user-defined location. 
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22. A medium as recited in claim 21 further cnveying 
software for inserting a pointing indicator adjacent a 
character in a serial data stream adjacent said 
nearest object. 

5 

23. A medium as recited in claim 20 further conveying 
browser software associated with said remote dis- 
play which is different browser software from 
browser software from a user location where said 
pointer indicator is generated. 10 
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